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A conceptual framework is provided for generating 
assessment tasks that provide an instructor with a richer description 
of student reasoning/thinking than is possible by simply giving 
students problems to solve. The application of this framework is 
illustrated with material from introductory statistics courses which 
focus on: (1) sampling; (2^ interval estimation; (3) point 
estimation; and (4) hypothesis testing. The authors adapt work done 
in mathematics by R # T. Putnam and others (199C) to the discipline of 
statistics. Statistical activities are related to a person's 
understanding of these activities. Examining these relationships 
provides a framework that helps guide statistics instructors in 
deciding which assessment tasks to create. The activities can be 
divided into: statistical problem solving; statistical modeling; and 
statistical argumentation. Five domains of cognition are defined: (1) 
understanding as representation; (?) understanding as appropriately 
integrated and organized knowledge structures; (3) understanding as 
connections between types of knowledge; (4) understanding as situated 
cognition; and (5) understanding as the active construction of 
knowledge. One table provides the framework for generating 
assessments by crossing activity domains with understanding domains. 
The use of MicroCAM, a microcomputer assisted measurement tool, is 
discussed within this framework. Two flowcharts illustrate pretest 
and posttest knowledge structures for a given student. (SLD) 
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Solving Problems is Not Enough: Assessing and Diagnosing 
the Ways in Which Students Organize Statistical Concepts 

Anthony J. Nitko and Suzanne Lane 
University of Pittsburgh 



Educators arc reformulating the goals of buth teaching and assessment within content 
domains. Some of the recurring themes include the need for instruction to promote students as 
thinkers, problem solvers, and inquirers: the need to conceptualize meaningful understanding as 
involving active construction: the need for testing for thinking and understanding; and the need 
for assessment tasks to more closely resemble "real world" learning tasks. According to this 
view, assessment procedures used in introductory statistics courses should not only assess 
students' skills in carrying out statistical procedures, but also their understanding and thinking 
about statistics. Teaching statistical thinking and reasoning would be gready facilitated, for 
example, by using instruments capable of assessing the structure and the relationships students 
imooc? on their own knowledge of statistical concepts. This paper provides a conceptual 
rramewonc for generating assessment tasks which provide an instructor with a richer description 
of students' thinking and reasoning than is possible by just giving students problems to solve. 
Although the framework is general, its application is illurtrated with material from college and 
beginning graduate level pre-caicuius introductory statistics courses and which focuses on the 
following topics: sampling, interval estimation, point estimation, and hypothesis testing. 

A fundamental objective of instruction should be the development of students' connectio*" s 
^mong concepts and The structuring of their knowledge tGlaser. 1989). A beginning student's 
knowledge base consists of isolated bits of information reflecting a shallow understanding of 
joncepts and their interrelationships. With instruction and experience, knowledge should become 
more structured and integrated with prior knowledge. 

Research thai examines how students process and structure knowledge has direct 
implications for tne design of more valuable assessments (Glaser, Lesgoid, & Lajoie, 1987: Lane. 
1989: Marshall, 1988: N'^.o, 1989). Tests need to assess not only declarative and procedural 
knowledge but also the relationships that provide the foundation for the organization of students' 
knowledge structures. There is a need to assess whether a student can obtain the right answer 
:o a problem and also whether the solutions are based on an understanding of underlying 
principles and concepts, that is, whether they are responding to the deep structure of the problem 
ratner than the problem's surface structure. Diagnosis of learning should focus on how a learner 
perceives the structure or organization of content and processes information and knowledge to 
soive problems (Nitko. 1989). Because students' structures of knowledge change over the course 
of instruction, such assessments should also be sensitive to modifications and reformulations in 
students' knowledge structures. 

In this paper we adapt work done in the mathematics discipline by Putnam. Lampert. and 
Peterson (1990) to the statistics discipline. We relate statistical activities to various ways of 
describing a person's understanding or cognition of these activities. The relationship between 
activities on the one hand and cognitions which underpin these activities on the other, provides 
a framework that gives statistics instructors guidance concerning the kinds of assessment tasks 



to create in order to assess the ways in which their students think about and reason with statistics. 
Statistical Activities 

Statistical activities may be divided into three interrelated domains: problem solving, 
statistical modeling, and statistical argumentation. Within each of these activity domains 
instructors can differentiate students in a statistics class. Sratistical problem solving is the ability 
to apply statistical concepts to problem situations. Students who are good problem solvers are 
able to abstract from a given situation the relevant bits of information and their relationships and 
to perform transformaaons on the mathematical relationships underlying symbols. Knowledge 
of conventions, symbols, and systems and their interrelationships underlie successful problem 
solving activity. Assessment tasks need to capture students' ability to formulate problems, 
identify principles underlying problems, analyze situations to determine common properties, 
verify and interpret results, and to generalize solutions. 

Statistical modeling involves knowing how to develop a model by representing the 
relationships in a problem graphically and symbolically. Students should also be able to describe 
the conditions under which a particular model is appropriate. Problem solving and statistical 
modeling are related by the activity of formulating a problem. Assessment tasks focus or. 
students' ability to identify appropriate models, distinguish among competing models, identify 
the conditions necessary for application of a certaui model, and identify how models are 
interrelated. 

Statistical argumentation is the ability to evaluate the truth of propositions and to develop 
plausible arguments tor statistical assertions. Inductive reasoning is required to recognize 
patterns and to make conjectures. Deductive reasoning is required to verify c inclusions, to judge 
;he validity of arguments by considering counter examples, and to construe valid arguments. 
Assessment tasks focus on students' ability to identify false arguments, to explain similarities and 
differences among competing arguments, and to araculate the relationships among concepts and 
procedures. 

Statistical Cognitions or Understandings 

Underpinning a person's performance in each of the statistical activity domains, is a 
complex array of cognitions wh en we generally describe as a person's understanding of statistics. 
We focus on five interrelated ways of describing a person's understanding which we call 
cognition domains: understanding as representation, understanding as knowledge structures, 
understanding as connections among types of knowledge, understanding as entailing active 
construcuon of knowledge, and understanding as situated cognition (Putnam, Lampert, & 
Peterson. 1990). 

Understanding as repre sentation means having internalized ideas, symbols, and systems 
and being able to move within and between them in ways that allow for successful problem 
solving activity. When assessing this aspect of understanding statistics, an instructor determines 
the relationships between an external portrayal of statistical information and the internal cognitive 
representations of a student. The representational ability of the student is reflected in the degree 
of organization in a student's knowledge structure (Glaser, 1989). Naive representations of 
subject matter represents only partial understanding by the student. Assessment tasks focus on 
student's ability to use statistical vocabulary, notation, and structure to represent ideas, solve 
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problems, and model situations. 



Understanding may be conceptualized as an appropriately integrated and organized 
knowledge structure. Understanding statistics also means having access to the knowledge and 
cognitive processes needed to perform various statistical tasks. Problem solving is intimately 
related to knowledge structures because coherent and organized knowledge structures underlie 
successful problem solving. Valuable assessments of students' thinking include examining the 
understandings and models that students construct for themselves during the learning process. 
Levels of achievement could be viewed as levels of understanding of concepts and their 
interrelationships that underlie a subject matter domain (Masters & Mislevy, in press). 
Assessment tasks capable of characterizing knowledge structures underlying problem solving 
activity provide valuable information about students' statistical understanding. 
Characteristics of knowledge structures that can be examined include the number and types of 
concepts depicted, the number and types of relationships among concepts (including conditional 
and hierarchial), and the degree of organization. For diagnostic purposes, congruence between 
a student's knowledge structure and various knowledge structures of experts may be examined. 
MicroCAM (described later in this paper) is one technique for assessing knowledge structures. 

Understanding mav be viewed as connections between different types of knowledge such 
as between conceptual and procedural knowledge and between formal, symbolic knowledge and 
informal knowledge. From this viewpoint, knowing statistics means understanding the concepts 
underlying the procedures and being fluent at integrating formal knowledge with informal 
knowledge developed outside of the class environment. Tasks focus on assessing an ability to 
create interrelationships across types of knowledge. 

Understanding as situated cognition means that understanding statistics involves the 
interplay between students' cognitive activity and physical and social situations. Learning is the 
process of constructing understanding out of activities embedded in the social and physical world. 
Knowledge may be viewed as a product of the acti /ity and situations in which they are produced 
and used (Brown. Collins, & Duguid, 1988). This involves students' participation in statistical 
activity or "doing'' statistics such as abstracting, convincing, inferring, organizing, representing, 
inventing, generalizing, explaining, validating, and conjecturing. Ecologically valid instructional 
and assessment tasks entail being representative of the ways in which knowledge and skills are 
used in "real world" contexts (Brown, Collins. & Duguid, 1989). This conceptualization of 
understanding implies the need for assessment tasks to be embedded in activities that have "real 
world" meaning. In a classroom assessment situation, these aenvities may be simulated by giving 
students background context, research questions, and sets of relevant and irrelevant raw data, and 
asking them to use the data to interpret and answer the research questions. 

Understanding as the active construcnon of knowledge means the process by which 
knowledge structures have been developed or acquired by students. Learning entails actively 
reorganizing and integrating new information with existing knowledge. Understanding statistics 
means having expanded one's own knowledge structures or ways of thinking to incorporate 
statistical concepts and principles. Assessment procedures need to be sensitive :o modifications 
in students' knowledge structures and requires the assessment of students at various points in 
time such as before and after instruction in order to trace developing knowledge structures. 
MicroCAM (described later) is an example of an assessment procedure that may be used to 
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examine changes in knowledge structures as a result of instruction. 

Generating Assessment Procedures and Tasks 
The foregoing theoretical conceptualizations of statistical activities and statistical 
understandings may be used to create a framework which gives guidance to statistics instructors 
for developing assessment procedures. Assessment tasks can be developed which measure 
students' levels of problem solving, modeling, and argumentation (reasoning) abilities using as 
a guide each of the cognition or understanding domains. This will ensure that an instructor 
obtains a comprehensive assessment of students' thinking that will be especially useful to guide 
instruction- 
It should be noted that the assessment framework we provide requires you to utilize a wide 
range of assessment techniques; many more techniques, in fact, than arc used for assessment in 
a typical introductory statistics course. Among the assessment techniques needed to adequately 
utilize our framework are the following question types: essays, shon answer, yes-no with student- 
supplied justifications, concept-orienied multiple-choice items, masterlxst, analogical reasoning, 
graphic inference, concept mapping, and computer guided Longer term individual and group 
projects may be required also. 

Table 1 provides a framework for generating assessment. Each of the three activity 
domains is crossed with three understanding domains. The body of the table contains examples 
of the types of student performances which an instructor may assess. As stated previously, the 
domains are interrelated so the table might be better conceptualized as seamless fabric, rather 
than as discrete cells. 

One way to view this table, is to read downward within a statistical activity type and 
across types of understanding. For example, within problem solving, you can see the wide range 
of performances a student should display if he or she understands the nature of a statistical 
problem and its solution. Simply obtaining the correct answer to a problem is insufficient to 
display deep understanding. 

We have illustrations of a variety of assessment tasks that correspond to each cell in the 
theoretical framework captured by Table L These tasks focus on sampling, interval estimation, 
point estimation, and hypothesis testing, content areas typically taught to majors in education or 
the social sciences in a college-level pre-calcuius introductory statistics course. Because of space 
limitations, these sample assessment tasks cannot be reproduced here. They may be obtained 
from the authors. 

Assessments in the cognition domains of active construction of knowledge and as situated 
cognition do not readily fit into the framework of Table 1, but these domains have been shown 
to underpin a students' understanding of problem solving, modeling, and argumentation (Lave, 
Smith, & Butler, 1988; Marshall, 1988; Schoenfeld 1988). A variety of techniques have been 
developed for assessing knowledge structures and their construction (for reviews see Nitko, 1989; 
Shavcison & Stanton, 1975). Here we discuss one technique, MicroCAM which uses a 
Macintosh computer. 



MicroCAM 

MicroCAM (Ju, 1989) is a microcomputer-assisted measurement tool that assesses 
students' knowledge structure and interrelationships among different types of knowledge in 
various subject matter domains. The program was developed in HyperTalk for the Macintosh 
microcomputer. Using MicroCAM, students provide a spatial display on the computer screen of 
their knowledge structures including the concepts, connections among concepts, and the 
relationships underlying the connections. Figure 1 shows one student's representation of concepts 
related to basic statistics. To obtain this figure, the student used a mouse to move the concepts 
in the boxes around the screen to depict how the student associated them. The student then 
connected the concepts by lines to show the linkages the student perceives. The student then 
entered words on the lines to describe the nature of the linkages or relationships. 

MicroCAM can be used to evaluate students' initial understanding prior to instruction as 
well as changes in understanding resulting from instruction. Understanding of students' initial 
organization of knowledge and the relationships underlying their knowledge structures allows the 
teacher to tailor instruction to promote individual student learning. Students' explicit 
representations of their knowledge structures prior to and after instruction provides a means to 
knowing students' misconceptions, weaknesses, or problems and this information may be used 
as a basis for remediation. Moreover, comparisons of students' knowledge strucrares before and 
after instruction can be used to determine the effectiveness of instruction. 

Teachers or researchers can develop their own assessment instruments using MicroCAM. 
the following types of assessments can be developed: (a) the teacher specifies concepts and 
presents a taxonomy of concept relations to the students, (b) the teacher specifies concepts, but 
does not present a taxonomy of relations to the student, (c) the teacher does not specify concepts, 
but does present a taxonomy of relations to the student, and (d) the teacher does not specify 
concepts, and does not present a taxonomy of relations to the students. The taxonomy of 
relations that was built into MicroCAM was derived from three sources ( Armbruster & Anderson. 
1984: BalLstaedt & Mandl, 1985: Fisher. 1988). Ju investigated MicroCAM's stability reliability, 
concurrent validity, and sensitivity to change during statistics instruction. Results were positive 
in each of these areas and lend support to the reliability and the validity of using MicroCAM to 
obtain useful information about students' knowledge structures. 

Implications for Integrating Assessment and Instruction 

The assessment framework presented in this paper can be a basis for providing valuable 
information regarding students' learning difficulties. Assessments developed using the ideas in 
Table 1 are capable of identifying misconceptions and missing concepts and links in students' 
knowledge provide information to guide the instruction. As another example, MicroCAM can 
be used prior to instruction to determine the statistical knowledge that students bring to a course 
including their misconceptions concepts and their interrelationships. Instruction can be aimed 
at building and organizing students' knowledge structures to facilitate statistical problem solving 
and reasoning. MicroCAM can also be used after instruction to assess the degree to 
whichstudents have organized and integrated new information with existing knowledge. 
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Table I Eramcwoik tor the Generation of the Assessment Tasks 



Domain of 
Understanding 

Satiates 



Problem Solving 



Domain of Statistical Activity 



Statistical Modeling 



Representation a. Formulate problems using stat- 
istical concepts and ideas 

b. Express solutions in terms of 
appropriate concepts, symbols and 
systems 

c. Translate words into statistical 
symbols and relationships and vice 
versa 

d. Use statistical vocabulary, notation, 
and structure to represent statistical 
problems, ideas, and situations 

Knowledge a. Identify principles that underlie 
Structure problems 

b. Organize and classify problems 



a. 



b. 



Use geometric, algebraic, and 
statistical models to represent 
data, problems and situations 
Describe how practical problems 
can be restated to fit one or more 
statistical models 
Identify data needed to support 
the fit of a statistical model 



Connections 
Among Types 
of Knowledge 



b. 



Explain bow statistical concepts 
are used to solve a problem 
Use statistical vocabulary, notation, 
and structure to describe relation- 
ships 

Analyze situations »o determine 
common properties, structures, ami 
patterns 



a. 



b. 
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Show how models are derived 
and interrelated 

Describe conditions under which 
statistical models are upheld 



Articulate systems or models, 
linking seve«"«l models into a 
unitary framework 
Articulate how statistical 
models may be used to answer 
research and practial questions 
Articulate the relationship of 
statistical modeling to oilier forms 
of knowledge acquisition 



Statistical Arguments 



a. Recognize false arguments 

b. Use inductive masoning to make 
and validate conjectures 

c. Critically evaluate conclusions 

and claims made using data obtained 
from samples and surveys 

d. Justify selection and use of 
specific statistical indices in 
particular research and/or decision 
contexts 

a. Explain similarities, differences, 
advantages, and disadvantages among 
competing statistical arguments 

b. Explain how statistical concepts are 
linked to specific statistical arguments 

a. Articulate the relationships among 
concepts, indices, and procedures 

b. Articulate relationships among formal 
and informal statistical knowledge 

c. Articulate how statistical arguments 

link to other epistomological frameworks 
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B. Student C Posttest 

Figure 1. Knowledge structures ot Student G., pre- and post-instruction. 
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